feat(import): simpler import + state restore logic#625
Merged
floryst merged 8 commits intostreaming-basefrom Mar 26, 2025
Merged
feat(import): simpler import + state restore logic#625floryst merged 8 commits intostreaming-basefrom
floryst merged 8 commits intostreaming-basefrom
Conversation
floryst
force-pushed
the
simplified-state-load-restore
branch
from
1849857 to
382c45e
Compare
floryst
force-pushed
the
streaming-base
branch
2 times, most recently
from
c7c1a32 to
19e8af5
Compare
floryst
force-pushed
the
simplified-state-load-restore
branch
from
382c45e to
f59f77d
Compare
floryst
force-pushed
the
streaming-base
branch
2 times, most recently
from
f854153 to
3c79e69
Compare
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using `evaluateChain` and `asyncSelect`. `evaluateChain` is responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source. To keep processing a data source like how the old pipeline code supported nested executions, `evaluateChain` is invoked inside a loop for every data source. `asyncSelect` is used to drive the loop execution, seleting `evaluateChain` promises whenever they are done. The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.
DataSource objects now represent a single type, rather than the bag-of-types done originally.
floryst
force-pushed
the
simplified-state-load-restore
branch
from
f59f77d to
ef98d95
Compare
feat(DataSource): simplified data structure
Contributor
Author
|
@PaulHax FYI I'm merging these changes into streaming-base and working off of that. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit contains two changes that probably can be split up into two separate PRs that can be applied to the base branch. I'm putting this up as a draft for now so @PaulHax can get a sense of the changes.
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using
evaluateChainandasyncSelect.evaluateChainis responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source.To keep processing a data source like how the old pipeline code supported nested executions,
evaluateChainis invoked inside a loop for every data source.asyncSelectis used to drive the loop execution, seletingevaluateChainpromises whenever they are done.The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.